compute: columnar Get/Mfp output via as_collection_core - #37763
Conversation
8dab0e8 to
2e5920e
Compare
2e5920e to
c420261
Compare
c420261 to
47a405c
Compare
0a3b022 to
570a36a
Compare
570a36a to
b01857d
Compare
b01857d to
176a1c1
Compare
176a1c1 to
489ab17
Compare
489ab17 to
4bcab91
Compare
4bcab91 to
00417d5
Compare
QA LLM Review1. MEDIUM -- Mfp/Get output messages are no longer capped at ~204 records, coarsening FlatMap's yield granularity
DetailsMeasured at this commit through
Two smaller consequences of the same size change: where the edge forks, The ~1.8 MiB ship threshold is |
00417d5 to
b74bdeb
Compare
|
Verified, and not fixed in this stack. Flagging it rather than patching it, because the fix changes an operator scheduling contract and should be a decision rather than a drive-by. The mechanism checks out. Neither option the review offers is free. Bounding this producer gives back the bulk containers the migration exists for, on every consumer, to fix a granularity problem in one of them. Making My preference is the resumable batch as a follow-up on top of the stack, the way #38368 came off the join review. Say the word and I will open it; if you would rather bound the producer first as a stopgap, that is a one-line threshold and I can measure what it costs the arrange path. Posted by Claude Code |
Make `as_collection_core` build its ok output into a `ConsolidatingColumnBuilder` and return a `CollectionEdge::Columnar`, flipping the Get and Mfp producers to emit the columnar edge. This is the first producer flip: the columnar arm goes live for Get/Mfp-fed edges, which Wave 1 already made every consumer accept natively or decode only at a sanctioned leaf. Rework the identity fast-path so an unarranged trivial MFP hands the input edge straight through (a columnar producer stays columnar with no `ColumnarToVec` hop) instead of forcing `as_specific_collection` to decode it to `Vec`. A keyed identity still reads its arrangement as a `Vec` edge. `as_specific_collection` stays the row-based consumer leaf. `ConsolidatingColumnBuilder` folds within-batch duplicates, matching the row-based `ConsolidatingContainerBuilder` this replaced, and emits a `Column` instead of a `Vec`. It stages owned `(Row, T, Diff)` tuples to consolidate in place, so the producer gives its records owned. `mfp_plan.evaluate` already produces a fresh owned `Row` per result, so this is a move into staging, not a new allocation. The borrowed-push no-owned-Row pattern is a consumer optimization for reading an existing columnar batch; it never applied to a producer computing new rows. The internal fueled `flat_map` chain (`CollectionBundle::flat_map`, `ArrangementFlavor::flat_map`, `flat_map_core_fallible`, `PendingWork:: do_work`) carried a `PushInto<(D, T, Diff)>` bound that constrained the element type up front. The bound was never exercised by those bodies (the caller's `logic` performs the `give`), so relax it to `DCB: ContainerBuilder` and drop the now-unused `D`, letting the producer choose its own builder and push shape through the same fuel machinery. Sink and `ensure_collections` temporal bucketing decode the produced edge at their existing sanctioned leaves. Tests: a `Get -> ArrangeBy` end-to-end unit test asserting the producer emits columnar and the arrange input keeps the columnar passthrough; an identity-fast-path test asserting the input edge variant is preserved; and a within-batch consolidation test asserting duplicate output rows fold to a single record with summed diff. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Give the missing-push-bound rationale a single owner, drop the comparisons against the row-based builder, and cut the test doc to what the asserts do not already state. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b74bdeb to
45cd71b
Compare
First producer flip: Get/Mfp emit their output as the columnar edge via
as_collection_core, building aConsolidatingColumnBuilderso within-batch consolidation matches the priorConsolidatingContainerBuilder.Columnar dataflow-edge migration. Design doc:
doc/developer/design/20260720_columnar_dataflow_edges.md(#37744).Part of CPU-51.